HADOOP-19857. Enable unit tests on GHA by pan3793 · Pull Request #8452 · apache/hadoop

pan3793 · 2026-04-22T08:57:59Z

Description of PR

This PR enables running unit tests in parallel on GitHub Actions, with ~150 test suites(classes) excluded due to flaky or consistently failing on GHA that need to be fixed later, all of the remaining test suites take ~2.5 hours to complete. As a comparison, Jenkins takes ~30 hours.

How was this patch tested?

GHA report.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (HADOOP-19857)?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

No AI usage.

pan3793 · 2026-04-22T09:05:13Z

+      options: --user ${{ needs.build-image.outputs.uid }}
+    strategy:
+      fail-fast: false
+      max-parallel: 6


note, this runs in the contributor's forked repo, it's free but has concurrency limitations, also remember that each split requires a build, larger parallel splits consume more overall time.

pan3793 · 2026-04-22T09:06:56Z

+        include:
+          - comment: hdfs
+            modules:
+              -pl :hadoop-hdfs


IIRC, this single module takes more than 4 hours to complete (even though more than 20 tests are excluded for now), we need to improve the slow tests and use tags to split it into more groups

okay, it actually takes 5 hours to complete (with ~30 tests excluded).

I split it into 2 groups:

slow, ~70 classes, which single test class takes more than 60s, all of them take ~150min

other, the rest of them

hope it can be completed in 2.5 hours next round

@ajfabbri, given the situation, I expect we may

exclude fewer than 200 test classes

keep others running stable on GHA, within 3 hours

not ready but almost here, with ~160 test suites (classes) excluded, all remaining tests run successfully in 2.5h

https://github.com/pan3793/hadoop/actions/runs/24870046901

I still need to re-run several rounds to ensure the initial test list runs stably.

@ajfabbri @steveloughran @slfan1989 would be great if you could take a look first.

@pan3793 Will look at it when I get back. I am on vacation until tomorrow night.

This seems fine. We have some work to do on improving tests: We need to make them faster, and completely eliminate flaky tests (ideally). This gives us an iterative way to tackle it. 👍

ajfabbri

Made an initial review of the current Draft. I had a couple of questions, but it looks good overall.

ajfabbri · 2026-04-29T21:06:22Z

+#
+
+# hadoop-common
+**/org/apache/hadoop/ipc/TestRPC.java


Which of these are just slow, and which tests are flaky / failures? Would be good to leave comments here and separate into sections, e.g.

# hadoop-hdfs ... # slow ... # flaky

for each project.

this list does not contain slow tests, only flaky or consistently failing cases

TBH, it will take a lot of effort to classify that, but it might not be worth doing. Contributors who want to improve it are easy to classify by running the test locally a few times, fixing a flaky or consistently failing test does not have much difference IMO

Agreed. Please add a comment at the top of the file, e.g. "Flaky and/or broken tests needing attention.". The intention is that we will fix or remove these over time. Slow tests are handled with the annotation. 👍

I updated the README.md under the same folder to explain that

ajfabbri · 2026-04-29T21:11:57Z

+        include:
+          - comment: hdfs
+            modules:
+              -pl :hadoop-hdfs


This seems fine. We have some work to do on improving tests: We need to make them faster, and completely eliminate flaky tests (ideally). This gives us an iterative way to tackle it. 👍

ajfabbri · 2026-04-29T21:17:02Z

 package org.apache.hadoop.oncrpc;

 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;


This needs to be a separate commit. When you are ready to merge, we can do an interactive rebase to squash all commits except this one, and add a descriptive log message to this commit.

I will split this PR into independent PRs, each of which focuses on one topic.

Either way is fine with me.

ajfabbri · 2026-04-29T21:21:06Z

 }

 @test "hadoop_stop_daemon_force_kill" {
+  skip "Skip on GitHub Actions now"


Why are we disabling this? Does this affect Yetus CI? We could do a conditional skip based on an environment variable which is only set for GH actions.

yea, need to change the condition to skip only on GHA.

ajfabbri · 2026-04-29T21:21:26Z

 # Verify that host directories get mounted without z option
 # and INFO messages get printed out
 @test "start-build-env.sh (Docker without z mount option)" {
+  skip "Skip on GitHub Actions now"


ajfabbri · 2026-04-29T21:21:33Z


 # Verify that host directories get mounted with z option
 @test "start-build-env.sh (Docker with z mount option)" {
+  skip "Skip on GitHub Actions now"


ajfabbri · 2026-04-29T21:25:35Z

-        <configuration>
-          <forkCount>0</forkCount>
-        </configuration>
-      </plugin>


Interesting. I agree we should fix the underlying issue. How can we make sure we catch "silent VM crashes" on jenkins?

It is ok if we don't have an answer, but should we start a mailing list thread, or file a JIRA? Just don't want to accidentally, silently, break a test.

I made this change last year, can't recall the details ... will revert it for now.

…-hbase-tests" This reverts commit d9f030c.

pan3793 · 2026-05-09T11:49:06Z

@ajfabbri @slfan1989 @steveloughran, this is ready for review. for stability, please check https://github.com/pan3793/hadoop/actions/workflows/build_and_test.yml?query=branch%3Agha-enable-tests

slfan1989 · 2026-05-11T05:58:52Z

@pan3793 Thank you, Pan, for your work on this. We’ve already discussed it offline, and I approve this PR.

github-actions Bot added YARN HDFS build trunk Common RBF Infra labels Apr 22, 2026

pan3793 force-pushed the gha-enable-tests branch from ee48d49 to 2f14ca3 Compare April 22, 2026 09:00

pan3793 commented Apr 22, 2026

View reviewed changes

Comment thread .github/workflows/tmpl_build_and_test.yml Outdated

pan3793 commented Apr 22, 2026

View reviewed changes

pan3793 force-pushed the gha-enable-tests branch 4 times, most recently from 8a5ffe4 to 1dcdec9 Compare April 23, 2026 02:34

apache deleted a comment from hadoop-yetus Apr 23, 2026

pan3793 force-pushed the gha-enable-tests branch 3 times, most recently from 1a874a0 to 5ee44de Compare April 24, 2026 11:40

apache deleted a comment from hadoop-yetus Apr 24, 2026

ajfabbri reviewed Apr 29, 2026

View reviewed changes

pan3793 added 2 commits May 8, 2026 15:59

test GITHUB_ACTIONS

d7960fb

Revert "Re-enable surefire fork on hadoop-yarn-server-timelineservice…

41a3758

…-hbase-tests" This reverts commit d9f030c.

github-actions Bot removed the YARN label May 8, 2026

pan3793 changed the title ~~[DO-NOT-MERGE] Enable tests on GHA~~ HADOOP-19857. Enable unit tests on GHA May 8, 2026

pan3793 marked this pull request as ready for review May 8, 2026 13:02

pan3793 added 4 commits May 9, 2026 00:52

empty

32b2993

empty

e289015

empty

f378799

empty

e303d00

pan3793 requested review from ajfabbri, slfan1989 and steveloughran May 9, 2026 08:24

empty

55bba11

pan3793 added 5 commits May 9, 2026 23:55

empty

bba94d6

empty

332e318

empty

2a6faa9

empty

7dcf294

empty

7bafc42

apache deleted a comment from hadoop-yetus May 11, 2026

slfan1989 approved these changes May 11, 2026

View reviewed changes

Conversation

pan3793 commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

Uh oh!

pan3793 Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajfabbri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajfabbri Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented May 9, 2026

Uh oh!

slfan1989 commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pan3793 commented Apr 22, 2026 •

edited

Loading

pan3793 Apr 22, 2026 •

edited

Loading

pan3793 Apr 22, 2026 •

edited

Loading

pan3793 Apr 22, 2026 •

edited

Loading

ajfabbri Apr 29, 2026 •

edited

Loading